System and method for field calibrating video and lidar subsystems using independent measurements

ABSTRACT

A system uses range and Doppler velocity measurements from a lidar subsystem and images from a video subsystem to estimate a six degree-of-freedom trajectory of a target. The video subsystem and the lidar subsystem may be aligned with one another, and hence calibrated, by determining, for example, a centroid of an iris determined from the lidar subsystem and a centroid of the iris determined from the video subsystem and determining a calibration offset between the two centroids.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of of U.S. application Ser. No.17/526,768, which was filed on Nov. 15, 2021, now granted as U.S. Pat.No. ______; which in turn is a continuation of U.S. application Ser. No.16/025,077, which was filed on Jul. 2, 2018, now granted as U.S. Pat.No. 11,181,625; which in turn is a continuation application of U.S.application Ser. No. 14/165,724, which was filed on Jan. 28, 2014, nowgranted as U.S. Pat. No. 10,018,711. Each of the foregoing applicationsis incorporated herein by reference.

FIELD OF THE INVENTION

The invention is generally related to calibrating video and lidarsubsystems with one another and more particularly, to calibrating thevideo and lidar subsystems using independent measurements from each ofthese systems.

BACKGROUND OF THE INVENTION

Obtaining accurate three-dimensional (3D) images of a target using atwo-dimensional (2D) video camera subsystem integrated with athree-dimensional lidar (laser radar) subsystem is difficult unless thetwo subsystems are calibrated with one another. Calibration may refer tothe registration of the two subsystems with one another such thatmeasurements of points on a surface made by the lidar subsystem areaccurately mapped to the same points as measured by the video subsystem(or substantially so within the measurement tolerances, etc., of eachsubsystem). In other words, accurate three-dimensional images requirecorrespondence between measurements of the 3D coordinates obtained fromthe lidar subsystem with the respective pixel positions in the 2D videoimage.

Factory calibration of the subsystems may be useful in obtaining aninitial registration between the lidar subsystem and the videosubsystem. However, because both the lidar subsystem and the videosubsystem employ various mechanical components, the factory calibrationmay change as a result of various relative mechanical changes occurringwithin the lidar subsystem as well as between the lidar subsystem andthe video subsystems, including, but not limited to mishandling,temperature fluctuations, mechanical strains, and other mechanicaleffects as would be appreciated. In particular, various movingmechanical parts included in or associated with the lidar subsystem(e.g., scanning mirrors and their alignment to one another and also tothe laser beams) are susceptible to such changes.

What is needed is an improved system and method for calibrating thevideo and lidar subsystems of a three-dimensional measurement system.

SUMMARY OF THE INVENTION

Various implementations of the invention combine measurements generatedby a a lidar system with images generated by a video system to resolve asix degrees of freedom trajectory that describes motion of a target.Once this trajectory is resolved, an accurate three-dimensional image ofthe target may be generated. In some implementations of the invention,the video subsystem and the lidar subsystem may be aligned with oneanother by mapping certain independent measurements of various featureson the target made by each of the subsystems to one another. In someimplementations, various components of the lidar subsystem (e.g.,scanning mirrors, etc.) are aligned with one another by mapping theindependent measurements of various features obtained by each of thecomponents to one another.

These implementations, their features and other aspects of the inventionare described in further detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a combined lidar and video camera system according tovarious implementations of the invention.

FIG. 2 illustrates a lidar (i.e., laser radar) according to variousimplementations of the invention.

FIG. 3 illustrates a scan pattern for a lidar subsystem that employs twolidar beams according to various implementations of the invention.

FIG. 4 illustrates a scan pattern for a lidar subsystem that employsfour lidar beams according to various implementations of the invention.

FIG. 5 illustrates a relationship between points acquired from the lidarsubsystem from separate beams at substantially the same instance of timethat may be used to estimate an x-component of angular velocity of atarget according to various implementations of the invention.

FIG. 6 illustrates a relationship between points acquired from the lidarsubsystem from separate beams at substantially the same instance of timethat may be used to estimate a y-component of angular velocity of atarget according to various implementations of the invention.

FIG. 7 illustrates a relationship between points acquired from the videosubsystem that may be used to estimate a two-dimensional (e.g. x and ycomponents) translational velocity and a z-component of angular velocityof a target according to various implementations of the invention.

FIG. 8 illustrates a scan pattern of a lidar beam according to variousimplementations of the invention.

FIG. 9 illustrates a timing diagram which may be useful for describingvarious timing aspects associated with measurements from the lidarsubsystem according to various implementations of the invention.

FIG. 10 illustrates a timing diagram which may be useful for describingvarious timing aspects associated with measurements from the lidarsubsystem in relation to measurements from the video subsystem accordingto various implementations of the invention.

FIG. 11 illustrates a block diagram useful for processing lidarmeasurements and video images according to various implementations ofthe invention.

FIG. 12 illustrates a block diagram useful for processing lidarmeasurements and video images according to various implementations ofthe invention.

FIG. 13 illustrates a block diagram useful for processing lidarmeasurements and video images according to various implementations ofthe invention.

FIG. 14 illustrates a block diagram useful for processing lidarmeasurements and video images according to various implementations ofthe invention.

FIG. 15 illustrates a block diagram useful for processing lidarmeasurements and video images according to various implementations ofthe invention.

FIG. 16 illustrates facial features monitored by the lidar subsystem andthe video subsystem according to various implementations of theinvention.

FIG. 17 illustrates an exemplary lidar-based face profile according tovarious implementations of the invention.

FIG. 18 illustrates a flowchart depicting example operations performedby a system for calibrating a lidar subsystem, according to variousaspects of the invention.

FIG. 19 illustrates a flowchart depicting example operations performedby a system for calibrating a lidar subsystem and a video subsystem,according to various aspects of the invention.

FIG. 20 illustrates an image from a video subsystem on which isoverlayed a number of points measured by a lidar subsystem in accordancewith various implementations of the invention.

FIG. 21 illustrates a relevant portion of the image illustrated in FIG.20 in further detail.

FIG. 22 illustrates an offset between a location of an iris based on theimage from the video subsystem and a location of the iris based on themeasurements from the lidar subsystem.

FIG. 23 illustrates an operation of a processing system to determinecalibration offsets between lidar subsystem 130 and video subsystem 150in accordance with various implementations of the invention.

DETAILED DESCRIPTION

FIG. 1 illustrates a combined lidar and video camera system 100 (orthree-dimensional measurement system 100) according to variousimplementations of the invention. Various implementations of theinvention utilize synergies between lidar measurements and video imagesto resolve six degrees of freedom for motion of a target to a degree nototherwise possible with either a lidar or video camera alone.

Combined system 100 includes a lidar subsystem 130, a video subsystem150, and a processing system 160. As illustrated, lidar subsystem 130includes two or more lidar beam outputs 112 (illustrated as a beam 112A,a beam 112B, a beam 112(n−1), and a beam 112 n); two or more reflectedbeam inputs 114 each corresponding to one of beams 112 (illustrated as areflected beam 114A, a reflected beam 114B, a reflected beam 114(n−1),and a reflected beam 114 n); two or more lidar outputs 116 eachassociated with a pair of beam 112/reflected beam 114 (illustrated as alidar output 116A associated with beam 112A/reflected beam 114A, a lidaroutput 116B associated with beam 112B/reflected beam 114B, a lidaroutput 116(n−1) associated with beam 112(n−1)/reflected beam 114(n−1),and a lidar output 116 n associated with beam 112 n/reflected beam 114n).

In some implementations of the invention, beam steering mechanism 140may be employed to steer one or more beams 112 toward target 190. Insome implementations of the invention, beam steering mechanism 140 mayinclude individual steering mechanisms, such as a steering mechanism140A, a steering mechanism 140B, a steering mechanism 140C, and asteering mechanism 140D, each of which independently steers a beam 112toward target 190. In some implementations of the invention, one beamsteering mechanism 140 may independently steer pairs or groups of beams112 toward target 190.

In some implementations of the invention, beam steering mechanism 140may include one or more mirrors, each of which may or may not beseparately controlled, each mirror steering one or more beams 112 towardtarget 190. In some implementations of the invention, beam steeringmechanism 140 may directly steer an optical fiber of beam 112 withoutuse of a mirror. In some implementations of the invention, beam steeringmechanism 140 may be controlled to steer beams 112 in azimuth and/orelevation. Various techniques may be used by beam steering mechanism 140to steer beam(s) 112 toward target 190 as would be appreciated.

In some implementations of the invention, beam steering mechanism 140may be used to control both an azimuth angle and an elevation angle oftwo beams 112 toward the target. By controlling both the azimuth angleand the elevation angle, the two beams 112 may be used to scan a volumefor potential targets or track particular targets such as target 190.Other scanning mechanisms may be employed as would be apparent. In someimplementations of the invention, the two beams 112 may be offset fromone another. In some implementations of the invention, the two beams 112may be offset vertically (e.g., in elevation) or horizontally (e.g., inazimuth) from one another by a predetermined offset and/or apredetermined angle, either of which may be adjustable or controlled.

In some implementations of the invention, beam steering mechanism 140may be used to control both an azimuth angle and an elevation angle offour beams 112 toward the target. In some implementations, the fourbeams 112 may be arranged with horizontal and vertical separations. Insome implementations, the four beams may be arranged so as to form atleast two orthogonal separations. In some implementations, the fourbeams may be arranged in a rectangular pattern, with pairs of beams 112offset from one another vertically and horizontally. In someimplementations, the four beams may be arranged in other patterns, withpairs of beams 112 offset from one another. The separations of the fourbeams 112 may be predetermined offsets and/or predetermined angles,which may be fixed, adjustable and/or controlled.

A certain portion of each beam 112 may be reflected back from target 190to lidar subsystem 130 as reflected beam 114. In some implementations ofthe invention and as illustrated in FIG. 1 , reflected beam 114 followsthe same optical path (though in reverse) as beam 112. In someimplementations of the invention, a separate optical path may beprovided in lidar subsystem 130 or in combined system 100 to accommodatereflected beam 114.

In some implementations of the invention, lidar subsystem 130 receives areflected beam 114 corresponding to each beam 112, processes reflectedbeam 114, and outputs lidar output 116 to processing system 160.

Combined system 100 also includes video subsystem 150. Video subsystem150 may include a video camera for capturing two dimensional images 155of target 190. Various video cameras may be used as would be apparent.In some implementations of the invention, the video camera may outputimages 155 as pixels at a particular resolution and at a particularimage or frame rate. Video images 155 captured by video subsystem 150are forwarded to processing system 160. In some implementations of theinvention, lidar subsystem 130 and video subsystem 150 are offset fromone another in terms of position and orientation. In particular, lidarmeasurements typically correspond to three dimensions (e.g., x, y, andz) whereas video images typically correspond to two dimensions (e.g., xand y). Various implementations of invention calibrate lidar subsystem130 with video subsystem 150 to ensure that data provided by each systemrefers to the same location in a given coordinate system as would beapparent.

Combined system 110 may include one or more optional video subsystems(not otherwise illustrated) for capturing additional two-dimensionalimages 155 of target 190 from different positions, perspectives orangles as would be apparent.

In some implementations of the invention, processing system 160 receiveslidar outputs 116 from lidar subsystem 130 and images 155 from videosubsystem 150 and stores them in a memory or other storage device 165for subsequent processing. Processing system 160 processes lidar outputs116 and images 155 to generate a three-dimensional image of target 190.In some implementations of the invention, processing system 160determines a trajectory of target 190 from a combination of lidaroutputs 116 and images 155 and uses the trajectory to generate a motionstabilized three-dimensional image of target 190.

In some implementations of the invention, lidar subsystem 130 mayinclude, for each of beams 112, a dual frequency, chirped coherent laserradar system capable of unambiguously and simultaneously measuring bothrange and Doppler velocity of a point on target 190. Such a laser radarsystem is described in U.S. Pat. No. 7,251,824, entitled “ChirpedCoherent Laser Radar System and Method,” (the “Chirped LidarSpecification”), which is incorporated herein by reference in itsentirety. For purposes of clarity, a “beam” referenced in the ChirpedLidar Specification is not the same as a “beam” referred to in thisdescription. More particularly, in the Chirped Lidar Specification, twobeams are described as output from the laser radar system, namely afirst beam having a first frequency (chirped or otherwise) and a secondbeam having a second frequency (chirped or otherwise) that aresimultaneously coincident on a point on a target to provide simultaneousmeasurements of both range and Doppler velocity of the point on thetarget. For purposes of simplicity and clarity, a singular “beam” asdiscussed herein may refer to the combined first and second beams outputfrom the laser radar system described in the Chirped LidarSpecification. The individual beams discussed in the Chirped LidarSpecification are referred to herein henceforth as “signals.”Nonetheless, various implementations of the invention may employ beamsother than those described in the Chirped Lidar Specification providedthese beams provide simultaneous range and Doppler velocity measurementsat points on the target.

FIG. 2 illustrates a lidar 210 that may be used to generate and processbeam 112 and reflected beam 114 to provide lidar output 116 according tovarious implementations of the invention. Each lidar 210 unambiguouslydetermines a range and Doppler velocity of a point on target 190relative to lidar 210. Lidar 210 includes a first frequency lidarsubsection 274 and a second frequency lidar subsection 276. Firstfrequency lidar subsection 274 emits a first frequency target signal 212toward target 190 and second frequency lidar subsection 276 emits asecond frequency target signal 214 toward target 190. The frequencies offirst target signal 212 and second target signal 214 may be chirped tocreate a dual chirp system.

First frequency lidar subsection 274 may include a laser sourcecontroller 236, a first laser source 218, a first optical coupler 222, afirst signal delay 244, a first local oscillator optical coupler 230,and/or other components. Second frequency lidar subsection 276 mayinclude a laser source controller 238, a second laser source 220, asecond optical coupler 224, a second signal delay 250, a second localoscillator optical coupler 232 and/or other components.

First frequency lidar subsection 274 generates first target signal 212and a first reference signal 242. First target signal 212 and firstreference signal 242 may be generated by first laser source 218 at afirst frequency that may be modulated at a first chirp rate. Firsttarget signal 212 may be directed toward a measurement point on target190 either independently or combined with second target signal 214.First frequency lidar subsection 274 may combine target signal 256 thatwas reflected from target 190 with first reference signal 242, which isdirected over a path with a known or otherwise fixed path length, toresult in a combined first target signal 262.

Second frequency lidar subsection 276 may be collocated and fixed withrespect to first frequency lidar subsection 274 (i.e., within lidar210). More particularly, the relevant optical components fortransmitting and receiving the respective laser signals may becollocated and fixed. Second frequency lidar subsection 276 may generatesecond target signal 214 and a second reference signal 248. Secondtarget signal 214 and second reference signal 248 may be generated bysecond laser source 220 at a second frequency that may be modulated at asecond chirp rate. In some implementations of the invention, the secondchirp rate is different from the first chirp rate.

Second target signal 214 may be directed toward the same measurementpoint on target 190 as first target beam 212. Second frequency lidarsubsection 276 may combine one portion of target signal 256 that wasreflected from target 190 with second reference signal 248, which isdirected over a path with a known or otherwise fixed path length, toresult in a combined second target signal 264.

Processor 234 receives combined first target signal 262 and combinedsecond target signal 264 and measures a beat frequency caused by adifference in path length between each of the reflected target signalsand its corresponding reference signal, and by any Doppler frequencycreated by target motion relative to lidar 210. The beat frequencies maythen be combined linearly to generate unambiguous determinations ofrange and Doppler velocity of target 190 as set forth in the ChirpedLidar Specification. In some implementations, processor 234 provides therange and Doppler velocity measurements to processing system 160. Insome implementations, processor 234 is combined with processing system160; in such implementations, processing system 160 receives combinedfirst target signal 262 and combined second target signal 264 and usesthem to determine range and Doppler velocity.

As described, each beam 112 provides simultaneous measurements of rangeand Doppler velocity of a point on target 190 relative to lidar 210.According to various implementations of the invention, various numbersof beams 112 may be used to provide these measurements of target 190. Insome implementations of the invention, two or more beams 112 may beused. In some implementations of the invention, three or more beams 112may be used. In some implementations of the invention four or more beams112 may be used. In some implementations of the invention, five or morebeams 112 may be used.

In various implementations of the invention, beams 112 may be used togather measurements for different purposes. For example, in someimplementations of the invention, a particular beam 112 may be used forpurposes of scanning a volume including target 190. In someimplementations of the invention, multiple beams 112 may be used toaccomplish such scanning. In some implementations of the invention, aparticular beam 112 may be used to monitor a particular feature orposition on target 190. In some implementations of the invention,multiple beams 112 may be used to independently monitor one or morefeatures and/or positions on target 190. In some implementations of theinvention, one or more beams 112 may be used to scan target 190 whileone or more other beams 112 may be used to monitor one or more featuresand/or positions on target 190.

In some implementations of the invention, one or more beams 112 may scantarget 190 to obtain a three dimensional image of target 190 while oneor more other beams 112 may be monitoring one or more features and/orpositions on target 190. In some implementations of the invention, aftera three dimensional image of target 190 is obtained, one or more beams112 may continue scanning target 190 to monitor and/or update the motionaspects of target 190 while one or more other beams 112 may monitor oneor more features and/or positions on target 110.

In some implementations of the invention, measurements obtained via oneor more beams 112 used to monitor and/or update the motion aspects oftarget 190 may be used to compensate measurements obtained via the oneor more other beams 112 used to monitor one or more features and/orpositions on target 190. In these implementations of the invention, thegross motion of target 190 may be removed from the measurementsassociated with various features and/or positions on target 190 toobtain fine motion of particular points or regions on target 190. Invarious implementations of the invention, fine motion of target 190 mayinclude various vibrations, oscillations, or motion of certain positionson the surface of target 190 relative to, for example, a center of mass,a center of rotation, another position on the surface of target 190 orother position. In various implementations of the invention, fine motionof target 190 may include, for example, relative motion of variousfeatures such as eyes, eyelids, lips, mouth corners, facial muscles ornerves, nostrils, neck surfaces, etc. or other features of target 190.

In some implementations of the invention, based on the gross motionand/or the fine motion of target 190, one or more physiologicalfunctions and/or physical activities of target 190 may be monitored. Forexample, U.S. Pat. No. 7,507,203, entitled “System and Method forRemotely Monitoring Physiological Functions” describes various systemsand methods for monitoring physiological functions and/or physicalactivities of an individual and is incorporated herein by reference inits entirety.

In some implementations of the invention, one or more beams 112 may beused to monitor one or more locations on an eyeball of target 190 andmeasure various position and motion aspects of the eyeball at the eachof these locations. Co-pending U.S. patent application Ser. No.11/610,867, entitled “System and Method for Tracking Eyeball Motion”describes various systems and methods for tracking the movement of aneyeball and is incorporated herein by reference in its entirety.

In some implementations of the invention, one or more beams 112 may beused to focus on various features or locations on a face of target 190and measure various aspects of the face with respect to the features orlocations on the face of target 190. For example, certain facialfeatures or facial expressions may be monitored over a period of time toinfer a mental state of target 190, to infer an intent of target 190, toinfer a deception level of target 190 or to predict an event associatedwith target 190 (e.g., certain facial muscles may twitch just prior to achange in expression or prior to speech).

In some implementations of the invention, one or more beams 112 may beused to monitor one or more locations on a neck of target 190. Themeasured motion aspects of the neck of target 190 may be used todetermine throat movement patterns, vocal cord vibrations, pulse rate,and/or respiration rate. In some implementations of the invention, oneor more beams 112 may be used to monitor one or more locations on anupper lip of target 190 to detect and measure vibrations associated withspeech of target 190. These vibrations may be used to substantiallyreproduce the speech of target 190.

In some implementations of the invention, one or more beams 112 mayserve one purpose during a first period or mode of operation of combinedsystem 100 and may switch to serve a different purpose during a secondperiod or mode of operation of combined system 100. For example, in someimplementations of the invention, multiple beams 112 may be used tomeasure various motion aspects of target 190 so that processing system160 may determine or acquire a trajectory of target 190. Once thetrajectory of target 190 is acquired, some of the multiple beams 112 mayswitch to monitoring certain other aspects or features of target 190while other ones of the multiple beams 112 measure motion aspects oftarget 190 so that its trajectory can be maintained.

In some implementations of the invention, five beams 112 scan target 190to obtain a three dimensional image of target 190. In theseimplementations, four of these beams 112 each scan a portion of target190 (using various scanning patterns as described in further detailbelow) while a fifth beam 112 performs an “overscan” of target 190. Theoverscan may be a circular, oval, elliptical or similar round scanpattern or a rectangular, square, diamond or similar scan pattern orother scan pattern useful for capturing multiple measurements of variouspoints on target 190 (or at least points within close proximity to oneanother) within relatively short time intervals. These multiplemeasurements may correspond to other measurements made by the fifth beam112 (i.e., multiple visits to the same point by the fifth beam 112) orto measurements made by one or more of the other four beams 112 (i.e.,visits to the same point by the fifth beam and one or more of the otherfour beams 112). In some implementations, the pattern of the overscanmay be selected to provide additional vertical and/or horizontal spreadbetween measurements of target 190. Both the multiple measurements andadditional spread may be used to improve estimates of the motion oftarget 190. Use of the fifth beam 112 to overscan target 190 may occurduring each of the different modes of operation referred to above.

In some implementations of the invention, once the trajectory of target190 is satisfactorily acquired, one or more beams 112 may providemeasurements useful for maintaining the trajectory of target 190 as wellas monitor other aspects of features of target 190. In suchimplementations, other beams 112 may be used to scan for other targetsin the scanning volume.

As illustrated in FIG. 1 , a target coordinate frame 180 may be used toexpress various measurements associated with target 190. Variouscoordinate frames may be used as would be appreciated. In someimplementations of the invention, various ones of the subsystems 130,150 may express aspects of target 190 in coordinate frames other thantarget coordinate frame 180 as would be appreciated. For example, insome implementations of the invention, a spherical coordinate frame(e.g., azimuth, elevation, range) may be used to express measurementsobtained via lidar subsystem 130. Also for example, in someimplementations of the invention, a two dimensional pixel-basedcoordinate frame may be used to express images 155 obtained via videosubsystem 150. Various implementations of the invention may use one ormore of these coordinate frames, or other coordinate frames, at variousstages of processing as will be appreciated.

As would be appreciated, in some implementations of the invention,various coordinate transformations may be required to transformmeasurements from lidar subsystem 130, which may be expressed in aspherical coordinates with reference to lidar subsystem 130 (sometimesreferred to as a lidar measurement space), to the motion aspects oftarget 190, which may be expressed in Cartesian coordinates withreference to target 190 (sometimes referred to as target space).Likewise, various coordinate transformations may be required totransform measurements from video subsystem 150, which may be expressedin Cartesian or pixel coordinates with reference to video subsystem 150(sometimes referred to as video measurement space), to the motionaspects of target 190. In addition, measurements from combined system100 may be transformed into coordinate frames associated with externalmeasurement systems such as auxiliary video, infrared, hyperspectral,multispectral or other auxiliary imaging systems. Coordinatetransformations are generally well known.

As would be appreciated, in some implementations of the invention,various coordinate transformations may be required to transformmeasurements from lidar subsystem 130 and/or video subsystem 150 toaccount for differences in position and/or orientation of each suchsubsystem 130, 150 as would be apparent.

FIG. 3 illustrates a scan pattern 300 which may be used to scan a volumefor targets 190 according to various implementations of the invention.Scan pattern 300 includes a first scan pattern section 310 and a secondscan pattern section 320. First scan pattern section 310 may correspondto a scan pattern of a first beam 112 (e.g., beam 112A) that may be usedto scan the volume (or portion thereof). Second scan pattern section 320may correspond to a scan pattern of a second beam 112 (e.g., beam 112B)that may be used to scan the volume (or portion thereof).

As illustrated in FIG. 3 , the first beam 112 scans an upper region ofscan pattern 300 whereas the second beam 112 scans a lower region ofscan pattern 300. In some implementations of the invention, the scanpattern sections 310, 320 may include an overlap region 330. Overlapregion 330 may be used to align or “stitch together” first scan patternsection 310 with second scan pattern section 320. In someimplementations of the invention, scan patterns 310, 320 do not overlapto form overlap region 330 (not otherwise illustrated).

In implementations of the invention where lidar subsystem 130 employs avertically displaced scan pattern 300 (such as that illustrated in FIG.3 ), first beam 112 is displaced vertically (i.e., by some verticaldistance, angle of elevation, or other vertical displacement) from asecond beam 112. In this way, the pair of beams 112 may be scanned witha known or otherwise determinable vertical displacement.

While scan pattern 300 is illustrated as having vertically displacedscan pattern sections 310, 320 in FIG. 3 , in some implementations ofthe invention, scan pattern may have horizontally displaced scansections. In implementations of the invention where lidar subsystem 130employs a horizontally displaced scan pattern (not otherwiseillustrated), first beam 112 is displaced horizontally (i.e., by somehorizontal distance, angle of azimuth, or other horizontal displacement)from second beam 112. In this way, the pair of beams 112 may be scannedwith a known or otherwise determinable horizontal displacement.

While FIG. 3 illustrates a scan pattern 300 with two verticallydisplaced scan pattern sections 310, 320, various numbers of beams maybe stacked to create a corresponding number of scan pattern sections aswould be appreciated. For example, three beams may be configured witheither vertical displacements or horizontal displacements to providethree scan pattern sections. Other numbers of beams may be used eitherhorizontally or vertically as would be appreciated.

FIG. 4 illustrates a scan pattern 400 for lidar subsystem 130 thatemploys four beams 112 according to various implementations of theinvention. As illustrated in FIG. 4 , lidar subsystem 130 includes fourbeams 112 arranged to scan a scan pattern 400. Scan pattern 400 may beachieved by having a first pair of beams 112 displaced horizontally fromone another and a second pair of beams 112 displaced horizontally fromone another and vertically from the first pair of beam 112, therebyforming a rectangular scanning arrangement. Other scanning geometriesmay be used as would be apparent. Scan pattern 400 may be achieved bycontrolling the beams independently from one another, as pairs (eitherhorizontally or vertically), or collectively, via beam scanningmechanism(s) 140.

Scan pattern 400 includes a first scan pattern section 410, a secondscan pattern section 420, a third scan pattern section 430, and a fourthscan pattern section 440. In some implementations of the invention, eachof the respective scan pattern sections 410, 420, 430, 440 may overlapan adjacent scan pattern portion by some amount (illustratedcollectively in FIG. 4 as overlap regions 450). For example, in someimplementations of the invention, scan pattern 400 includes an overlapregion 450 between first scan pattern section 410 and third scan patternsection 430. Likewise, an overlap region 450 exists between a first scanpattern section 410 and a second scan section 420. In someimplementations of the invention, various ones of these overlap regions450 may not occur or otherwise be utilized. In some implementations ofthe invention, for example, only vertical overlap regions 450 may occuror be utilized. In some implementations of the invention, onlyhorizontal overlap regions 450 may occur or be utilized. In someimplementations of the invention, no overlap regions 450 may occur or beutilized. In some implementations of the invention, other combinationsof overlap regions 450 may be used.

As illustrated in FIG. 3 and FIG. 4 , the use by lidar subsystem 130 ofmultiple beams 112 may increase a rate at which a particular volume (orspecific targets within the volume) may be scanned. For example, a givenvolume may be scanned twice as fast using two beams 112 as opposed toscanning the same volume with one beam 112. Similarly, a given volumemay be scanned twice as fast using four beams 112 as opposed to scanningthe same volume with two beams 112, and four times as fast as scanningthe same volume with one beam 112. In addition, multiple beams 112 maybe used to measure or estimate various parameters associated with themotion of target 190 as will be discussed in more detail below.

According to various implementations of the invention, particular scanpatterns (and their corresponding beam configurations) may be used toprovide measurements and/or estimates of motion aspects of target 190.As described above, each beam 112 may be used to simultaneously providea range measurement and a Doppler velocity measurement at each pointscanned.

In some implementations of the invention, for each beam 112, a pointscanned by that beam 112 may be described by an azimuth angle, anelevation angle, and a time. Each beam 112 provides a range measurementand a Doppler velocity measurement at that point and time. In someimplementations of the invention, each point scanned by beam 112 may beexpressed as an azimuth angle, an elevation angle, a range measurement,a Doppler velocity measurement, and a time. In some implementations ofthe invention, each point scanned by beam 112 may be expressed inCartesian coordinates as a position (x, y, z), a Doppler velocity and atime.

According to various implementations of the invention, measurements fromlidar subsystem 130 (i.e., lidar outputs 116) and measurements fromvideo subsystem 150 (frames 155) may be used to measure and/or estimatevarious orientation and/or motion aspects of target 190. Theseorientation and/or motion aspects of target 190 may include position,velocity, acceleration, angular position, angular velocity, angularacceleration, etc. As these orientation and/or motion aspects aremeasured and/or estimated, a trajectory of target 190 may be determinedor otherwise approximated. In some implementations of the invention,target 190 may be considered a rigid body over a given time interval andits motion may be expressed as translational velocity componentsexpressed in three dimensions as v_(x) ^(trans), v_(y) ^(trans), andv_(z) ^(trans), and angular velocity components expressed in threedimensions as ω_(x), ω_(y), and ω_(z) over the given time interval.Collectively, these translational velocities and angular velocitiescorrespond to six degrees of freedom of motion for target 190 over theparticular time interval. In some implementations of the invention,measurements and/or estimates of these six components may be used toexpress a trajectory for target 190. In some implementations of theinvention, measurements and/or estimates of these six components may beused to merge the three-dimensional image of target 190 obtained fromlidar subsystem 130 with the two-dimensional images of target 190obtained from video subsystem 150 to generate three-dimensional videoimages of target 190.

In some implementations of the invention, the instantaneous velocitycomponent v_(z)(t) of a point on target 190 may be calculated based onthe range measurement, the Doppler velocity measurement, the azimuthangle and the elevation angle from lidar subsystem 130 as would beapparent.

Lidar subsystem 130 may be used to measure and/or estimate translationalvelocity v_(z) ^(trans) and two angular velocities of target 190, namelyω_(x) and ω_(y). For example, FIG. 5 illustrates an exemplaryrelationship between points with corresponding measurements from twobeams 112 that may be used to estimate x and y components of angularvelocity of target 190 according to various implementations of theinvention. More particularly, and generally speaking, as illustrated inFIG. 5 , in implementations where beams 112 are displaced from oneanother along the y-axis, a local velocity along the z-axis of pointP_(A) determined via a first beam 112, a velocity of point P_(B)determined via a second beam 112, and a distance between P_(A) and P_(B)may be used to estimate an angular velocity of these points about thex-axis (referred to herein as ω_(x)) as would be appreciated. In someimplementations of the invention, these measurements may be used toprovide an initial estimate of ω_(x).

FIG. 6 illustrates another exemplary relationship between points withcorresponding measurements from two beams 112 that may be used toestimate an angular velocity according to various implementations of theinvention. More particularly, as illustrated in FIG. 6 , inimplementations were beams 112 are displaced from one another on target190 along the x-axis, a velocity of point P_(A) determined via a firstbeam 112, a velocity of point P_(B) determined by a second beam 112, anda distance between P_(A) and P_(B) on target 190 along the x-axis may beused to estimate an angular velocity of these points about the y-axis(referred to herein as ω_(y)). In some implementations of the invention,these measurements may be used to provide an initial estimate of ω_(y).

FIG. 5 and FIG. 6 illustrate implementations of the invention where twobeams 112 are disposed from one another along a vertical axis or ahorizontal axis, respectively, and the corresponding range (which may beexpressed in three-dimensional coordinates x, y, and z) and Dopplervelocity at each point are measured at substantially the same time. Inimplementations of the invention that employ beams 112 along a singleaxis (not otherwise illustrated), an angular velocity may be estimatedbased on Doppler velocities measured at different points at differenttimes along the single axis. As would be appreciated, better estimatesof angular velocity may obtained using: 1) measurements at points at theextents of target 190 (i.e., at larger distances from one another), and2) measurements taken within the smallest time interval (so as tominimize any effects due to acceleration).

FIG. 5 and FIG. 6 illustrate conceptual estimation of the angularvelocities about different axes, namely the x-axis and the y-axis. Ingeneral terms, where a first beam 112 is displaced on target 190 along afirst axis from a second beam 112, an angular velocity about a secondaxis orthogonal to the first axis may be determined from the velocitiesalong a third axis orthogonal to both the first and second axes at eachof the respective points.

In some implementations of the invention, where two beams are displacedalong the y-axis from one another (i.e., displaced vertically) andscanned horizontally with vertical separation between scans, estimatesof both ω_(x) and ω_(y) may be made. While simultaneous measurementsalong the x-axis are not available, they should be sufficiently close intime in various implementations to neglect acceleration effects. In someimplementations of the invention where two beams 112 are displaced alongthe x-axis from one another and at least a third beam 112 is displacedalong the y-axis from the pair of beams 112, estimates of ω_(x), ω_(y)and v_(z) ^(trans) may be made. In some implementations of theinvention, estimates of both ω_(x), ω_(y) and v_(z) ^(trans) may be madeusing four beams 112 arranged in a rectangular fashion. In suchimplementations, the measurements obtained from the four beams 112include more information than necessary to estimate ω_(x), ω_(y) andv_(z) ^(trans). This so-called “overdetermined system” may be used toimprove the estimates of ω_(x), ω_(y) and v_(z) ^(trans) as would beappreciated.

As has been described, range and Doppler velocity measurements taken atvarious azimuth and elevation angles and at various points in time bylidar subsystem 130 may be used to estimate translational velocity v_(z)^(trans) and estimate two angular velocities, namely, cox and ω_(y), forthe rigid body undergoing ballistic motion.

In some implementations of the invention, ω_(x), ω_(y) and v_(z)^(trans) may be determined at each measurement time from themeasurements obtained at various points as would be appreciated. In someimplementations of the invention, ω_(x), ω_(y) and v_(z) ^(trans) may beassumed to be constant over an particular interval of time. In someimplementations of the invention, ω_(x), ω_(y) and v_(z) ^(trans) may bedetermined at various measurement times and subsequently averaged over aparticular interval of time to provide estimates of ω_(x), ω_(y) andv_(z) ^(trans) for that particular interval of time as would beappreciated. In some implementations of the invention, the particulartime interval may be fixed or variable depending, for example, on themotion aspects of target 190. In some implementations of the invention,a least squares estimator may be used to provide estimates of ω_(x),ω_(y) and v_(z) ^(trans) over a particular interval of time as would beappreciated. Estimates of ω_(x), ω_(y) and v_(z) ^(trans) may beobtained in other manners as would be appreciated.

In some implementations of the invention, images from video subsystem150 may be used to estimate three other motion aspects of target 190,namely translational velocity components v_(x) ^(trans) and v_(y)^(trans) and angular velocity component ω_(z) over a given interval oftime. In some implementations of the invention, frames 155 captured byvideo subsystem 150 may be used to estimate x and y components ofvelocity for points on target 190 as it moves between frames 155. FIG. 7illustrates a change in position of a particular point or feature I_(A)between a frame 155 at time T and a frame 155 at subsequent time T+Δt.

In some implementations of the invention, this change of position isdetermined for each of at least two particular points or features inframe 155 (not otherwise illustrated). In some implementations of theinvention, the change of position is determined for each of many pointsor features. In some implementations of the invention, translationalvelocity components v_(x) ^(trans) and v_(y) ^(trans), and angularvelocity component ω_(z) of target 190 may be estimated based on adifference in position of a feature I_(A)(T) and I_(A)(T+Δt) and adifference in time, Δt, between the frames 155. These differences inposition and time may be used to determine certain velocities of thefeature, namely, v_(x) ^(feat) and v_(y) ^(feat) that may in turn beused to estimate the translational velocity components v_(x) ^(trans)and v_(y) ^(trans), and angular velocity component ω_(z) of target 190.Such estimations of velocity and angular velocity of features betweenimage frames are generally understood as would be appreciated.

In some implementations of the invention, many features of target 190are extracted from consecutive frames 155. The velocities v_(x) ^(feat)and v_(y) ^(feat) of these features over the time interval betweenconsecutive frames 155 may be determined based on changes in position ofeach respective feature between the consecutive frames 155. A leastsquares estimator may be used to estimate the translational velocitiesv_(x) ^(trans) and v_(y) ^(trans), and the angular velocity ω_(z) fromthe position changes of each the extracted features.

In some implementations of the invention, a least squares estimator mayuse measurements from lidar subsystem 130 and the changes in position ofthe features in frames 155 from video subsystem 150 to estimate thetranslational velocities v_(x) ^(trans), v_(y) ^(trans) and v_(z)^(trans) and the angular velocities ω_(x), ω_(y), and ω_(z) of target190.

As has been described above, lidar subsystem 130 and video subsystem 150may be used to estimate six components that may be used describe themotion of target 190. These components of motion may be collected overtime to calculate a trajectory of target 190. This trajectory may thenbe used to compensate for motion of target 190 to obtain a motionstabilized three dimensional image of target 190. In variousimplementations of the invention, the trajectory of target 190 may beassumed to represent ballistic motion over various intervals of time.The more accurately trajectories of target 190 may be determined, themore accurately combined system 100 may adjust the measurements oftarget 190 to, for example, represent three dimensional images, or otheraspects, of target 190.

In various implementations of the invention, a rate at whichmeasurements are taken by lidar subsystem 130 is different from a rateat which frames 155 are captured by video subsystem 150. In someimplementations of the invention, a rate at which measurements are takenby lidar subsystem 130 is substantially higher than a rate at whichframes 155 are captured by video subsystem 150. In addition, becausebeams 112 are scanned through a scan volume by lidar subsystem 130,measurements at different points in the scan volume may be taken atdifferent times from one another; whereas pixels in a given frame 155are captured substantially simultaneously (within the context of videoimaging). In some implementations of the invention, these timedifferences are resolved in order to provide a more accurate trajectoryof target 190.

As illustrated in FIG. 8 , in some implementations of the invention, ascan pattern 840 may be used to scan a volume for targets. For purposesof explanation, scan pattern 840 represents a pattern of measurementstaken by a single beam. In some implementations multiple beams may beused, each with their corresponding scan pattern as would be apparent.As illustrated, scan pattern 840 includes individual points 810 measuredleft to right in azimuth at a first elevation 831, right to left inazimuth at a second elevation 832, left to right in azimuth at a thirdelevation 833, etc., until a particular scan volume is scanned. In someimplementations, scan pattern 840 may be divided into intervalscorresponding to various timing aspects associated with combined system100. For example, in some implementations of the invention, scan pattern840 may be divided into time intervals associated with a frame rate ofvideo subsystem 150. In some implementations of the invention, scanpattern 840 may be divided into time intervals associated with scanninga particular elevation (i.e., an entire left-to-right or right-to-leftscan). In some implementations of the invention, scan pattern 840 may bedivided into time intervals associated with a roundtrip scan 820(illustrated in FIG. 8 as a roundtrip scan 820A, a roundtrip scan 820B,and a roundtrip scan 820C) at one or more elevations (i.e., aleft-to-right and a return right-to-left scan at either the same ordifferent elevations). Similar timing aspects may be used inimplementations that scan vertically in elevation (as opposed tohorizontally in azimuth). Other timing aspects may be used as well.

As illustrated in FIG. 8 and again for purposes of explanation, eachinterval may include N points 810 which may in turn correspond to thenumber of points 810 in a single scan (e.g., 831, 832, 833, etc.) or ina roundtrip scan 820. A collection of points 810 for a particularinterval is referred to herein as a sub-point cloud and a collection ofpoints 810 for a complete scan pattern 840 is referred to herein as apoint cloud. In some implementations of the invention, each point 810corresponds to the lidar measurements of range and Doppler velocity at aparticular azimuth, elevation, and a time at which the measurement wastaken. In some implementations of the invention, each point 810corresponds to the lidar measurements of range (expressed x, y, zcoordinates) and Doppler velocity and a time at which the measurementwas taken.

FIG. 9 illustrates a timing diagram 900 useful for describing varioustiming aspects associated with measurements from lidar subsystem 130according to various implementations of the invention. Timing diagram900 includes points 810 scanned by beam 112, sub-point clouds 920 formedfrom a plurality of points 810 collected over an interval correspondingto a respective sub-point cloud 920, and a point cloud 930 formed from aplurality of sub-point clouds 920 collected over the scan pattern.Timing diagram 900 may be extended to encompass points 810 scanned bymultiple beams 112 as would be appreciated.

Each point 810 is scanned by a beam 112 and measurements associated witheach point 810 are determined by lidar subsystem 130. In someimplementations of the invention, points 810 are scanned via a scanpattern (or scan pattern section). The interval during which lidarsubsystem 130 collects measurements for a particular sub-point cloud 920may have a time duration referred to as T_(SPC). In some implementationsof the invention, the differences in timing of the measurementsassociated with individual points 810 in sub-point cloud 920 may beaccommodated by using the motion aspects (e.g., translational velocitiesand angular velocities) for each point to adjust that point to aparticular reference time for sub-point cloud 920 (e.g., t_(RSPC)). Thisprocess may be referred to as stabilizing the individual points 810 forthe motion aspects of target 190.

In some implementations of the invention, the velocities may be assumedto be constant over the time interval (i.e., during the time durationT_(SPC)). In some implementations of the invention, the velocities maynot be assumed to be constant during the period of the scan pattern andacceleration effects may need to be considered to adjust themeasurements of points 810 to the reference time as would beappreciated. In some implementations of the invention, adjustments dueto subdivision of the time interval may also need to be accommodated. Asillustrated in FIG. 9 , the reference time for each sub-point cloud 920may be selected at the midpoint of the interval, although otherreference times may be used.

In some implementations of the invention, similar adjustments may bemade when combining sub-point clouds 920 into point clouds 930. Moreparticularly, in some implementations of the invention, the differencesin timing of the measurements associated with sub-point clouds 920 inpoint cloud 930 may be accommodated by using the motion aspectsassociated with the measurements.

In some implementations of the invention, the measurements associatedwith each sub-point cloud 920 that is merged into point cloud 930 areindividually adjusted to a reference time associated with point cloud930. In some implementations of the invention, the reference timecorresponds to a frame time (e.g., time associated with a frame 155). Inother implementations of the invention, the reference time correspond toan earliest of the measurement times of points 1110 in point cloud 930,a latest of the measurement times of points 1110 in point cloud 930, anaverage or midpoint of the measurement times of points 1110 in pointcloud 930, or other reference time associated with point cloud 930.

Although not otherwise illustrated, in some implementations of theinvention, similar adjustments may be made to combine point clouds 930from individual beams 112 into aggregate point clouds at a particularreference time. In some implementations of the invention, this may beaccomplished at the individual point level, the sub-point cloud level orthe point cloud level as would be appreciated. For purposes of theremainder of this description, sub-point clouds 920 and point clouds 930refer to the collection of points 810 at their respective referencetimes from each of beams 112 employed by lidar subsystem 130 to scantarget 190.

In some implementations of the invention, motion aspects of target 190may be assumed to be constant over various time intervals. For example,motion aspects of target 190 may be assumed to be constant over T_(SPC)or other time duration. In some implementations of the invention, motionaspects of target 190 may be assumed to be constant over a givenT_(SPC), but not necessarily constant over T_(PC). In someimplementations of the invention, motion aspects of target 190 may beassumed to be constant over incremental portions of T_(SPC), but notnecessarily over the entire T_(SPC). As a result, in someimplementations of the invention, a trajectory of target 190 may beexpressed as a piece-wise function of time, with each “piece”corresponding to the motion aspects of target 190 over each individualtime interval.

In some implementations, timing adjustments to compensate for motion maybe expressed as a transformation that accounts for the motion of a pointfrom a first time to a second time. This transformation, when applied tomeasurements from, for example, lidar subsystem 130, may perform thetiming adjustment from the measurement time associated with a particularpoint (or sub-point cloud or point cloud, etc.) to the desired referencetime. Furthermore, when the measurements are expressed as vectors, thistransformation may be expressed as a transformation matrix. Suchtransformation matrices and their properties are generally well known.

As would be appreciated, the transformation matrices may be readily usedto place a position and orientation vector for a point at any time to acorresponding position and orientation vector for that point at anyother time, either forwards or backwards in time, based on the motion oftarget 190. The transformation matrices may be applied to sub-pointclouds, multiple sub-point clouds and point clouds as well. In someimplementations, a transformation matrix may be determined for eachinterval (or subinterval) such that it may be used to adjust a pointcloud expressed in one interval to a point cloud expressed in the nextsequential interval. In these implementations, each interval has atransformation matrix associated therewith for adjusting the pointclouds for the trajectory of target 190 to the next interval. In someimplementations, a transformation matrix may be determined for eachinterval (or subinterval) such that it may be used to adjust a pointcloud expressed in one interval to a point cloud expressed in the priorsequential interval. Using the transformation matrices for variousintervals, a point cloud can be referenced to any time, either forwardor backward.

FIG. 10 illustrates a timing diagram 1000 useful for describing varioustiming aspects associated with measurements from lidar subsystem 130 inrelation to measurements from video subsystem 150 according to variousimplementations of the invention. In some implementations of theinvention, point cloud 930 may be referenced to the midpoint of a timeinterval between frames 155 or other time between frames 155. In someimplementations of the invention, point cloud 930 may be referenced to aframe time corresponding to a particular frame 155. Point cloud 930 maybe referenced in other manners relative to a particular frame 155 aswould be appreciated.

As illustrated in FIG. 10 , PC_(m−1) is the expression of point cloud930 referenced at the frame time of frame I_(n−1); PC_(m) is theexpression of point cloud 930 referenced at the frame time of frameI_(n); and PC_(m+1) is the expression of point cloud 930 referenced atthe frame time of frame I_(n+1); and PC_(m+2) is the expression of pointcloud 930 referenced at the frame time of frame I_(n+2). In someimplementations, point cloud 930 may be referenced at other times inrelation to the frames and frames times as would be apparent.

As described above, a transformation matrix T_(i,i+1) may be determinedto transform an expression of point cloud 930 at the i^(th) frame timeto an expression of point cloud 930 at the (i+1)^(th) frame time. Inreference to FIG. 10 , a transformation matrix T_(m−1,m) may be used totransform PC_(m−1) to PC_(m); a transformation matrix T_(m,m+1) may beused to transform PC_(m) to PC_(m+1); and a transformation matrixT_(m+1,m+2) may be used to transform PC_(m+1) to PC_(m+2). In this way,transformation matrices may be used to express point clouds 930 atdifferent times corresponding to frames 155.

According to various implementations of the invention, thetransformation matrices which are applied to point cloud 930 to expresspoint cloud 930 from a first time to a second time are determined indifferent processing stages. Generally speaking, transformation matricesare directly related with six degree of motion parameters ω_(x), ω_(y),ω_(z), v_(x) ^(trans), v_(y) ^(trans) and v_(z) ^(trans) that may becalculated in two steps: first ω_(x), ω_(y), and v_(z) ^(trans) fromlidar subsystem and second v_(x) ^(trans), v_(y) ^(trans) and ω_(z),from video subsystem.

FIG. 11 illustrates a block diagram of a configuration of processingsystem 160 that may be used during a first phase of the first processingstage to estimate a trajectory of target 190 according to variousimplementations of the invention. In some implementations of theinvention, during the first phase of the first stage, a series ofinitial transformation matrices (referred to herein as T_(i,i−1) ⁽⁰⁾)are determined from various estimates of the motion aspects of target190. As illustrated, lidar subsystem 130 provides range, Dopplervelocity, azimuth, elevation and time for at each point as input to aleast squares estimator 1110 that is configured to estimate angularvelocities cox and ω_(y) and translational velocity v_(z) ^(trans) overeach of a series of time intervals. In some implementations of theinvention, angular velocities ω_(x) and ω_(y) and translational velocityv_(z) ^(trans) are iteratively estimated by varying the size of the timeintervals (or breaking the time intervals into subintervals) asdiscussed above until any residual errors from least squares estimator1110 for each particular time interval reach acceptable levels as wouldbe apparent. This process may be repeated for each successive timeinterval during the time measurements of target 190 are taken by lidarsubsystem 130.

Assuming that target 190 can be represented over a given time intervalas a rigid body (i.e., points on the surface of target 190 remain fixedwith respect to one another) undergoing ballistic motion (i.e., constantvelocity with no acceleration), an instantaneous velocity of any givenpoint 810 on target 190 can be expressed as:

v=v ^(trans)[ω×(R−R _(c) −v ^(trans) *Δt)]  Eq. (1)

where

-   -   v is the instantaneous velocity vector of the given point;    -   v^(trans) is the translational velocity vector of the rigid        body;    -   ω is the rotational velocity vector of the rigid body;    -   R is the position of the given point on the target;

R_(c) is the center of rotation for the target; and

-   -   Δt is the time difference of each measurement time from a given        reference time.

Given the measurements available from lidar subsystem 130, thez-component of the instantaneous velocity may be expressed as:

v _(z) =v _(z) ^(trans)[ω×(R−R _(c) −v _(trans) *Δt)]_(z)  Eq. (2)

where

-   -   v_(z) is the z-component of the instantaneous velocity vector;    -   v_(z) ^(trans) is the z-component of the translational velocity        vector; and    -   [ω×(R−R_(c)−v^(trans)*Δt)]_(z) is the z-component of the cross        product.

In some implementations of the invention, frame-to-frame measurementscorresponding to various features from images 155 may be made. Thesemeasurements may correspond to a position (e.g., x^(feat), y^(feat)))and a velocity (e.g., v_(x) ^(feat), v_(y) ^(feat)) for each of thefeatures and for each frame-to-frame time interval. In implementationswhere a z-coordinate of position is not available from video subsystem150, an initial estimate of z may be made using, for example, an averagez component from the points from lidar subsystem 130. Least squaresestimator 1120 estimates angular velocities ω_(x), ω_(y), and ω_(z) andtranslational velocities v_(x) ^(trans), v_(y) ^(trans), and v_(z)^(trans) which may be expressed as a transformation matrix T_(i,i+1) ⁽⁰⁾for each of the relevant time intervals. In some implementations of theinvention, a cumulative transformation matrix corresponding to thearbitrary frame to frame time interval may be determined.

FIG. 12 illustrates a block diagram of a configuration of processingsystem 160 that may be used during a second phase of the firstprocessing stage to estimate a trajectory of target 190 according tovarious implementations of the invention. In some implementations of theinvention, during the second phase of the first stage, newtransformation matrices (referred to herein as T_(i,i+1) ⁽¹⁾) aredetermined from various estimates of the motion aspects of target 190.As illustrated, measurements from lidar subsystem 130 of range, Dopplervelocity, azimuth, elevation and time for at each of the N points areinput to a least squares estimator 1110 of processing system 160 alongwith the transformation matrices T_(i,i+1) ⁽⁰⁾ to estimate angularvelocities cox and ω_(y) and translational velocity v_(z) ^(trans) overeach of a series of time intervals in a manner similar to that describedabove during the first phase.

The primary difference between the second phase and the first phase isthat least squares estimator 1120 uses the calculated z position of thefeatures based on T_(i,i+1) ⁽⁰⁾ as opposed to merely an average of zposition. Least squares estimator 1120 estimates new angular velocitiesω_(x), ω_(y), and ω_(z) and new translational velocities v_(x) ^(trans),v_(y) ^(trans), and v_(z) ^(trans) which may be expressed as atransformation matrix T_(i,i+1) ⁽¹⁾ for each of the relevant timeintervals. Again, in some implementations of the invention, a cumulativetransformation matrix corresponding to the frame to frame time intervalmay be determined.

FIG. 13 illustrates a block diagram of a configuration of processingsystem 160 that may be used during a third phase of the first processingstage to estimate a trajectory of target 190 according to variousimplementations of the invention. In some implementations of theinvention, during the third phase of the first stage, new transformationmatrices (referred to herein as T_(i,i+1) ⁽²⁾) are determined fromvarious estimates of the motion aspects of target 190. As illustrated,lidar subsystem 130 provides range, Doppler velocity, azimuth, elevationand time for at each of the points as input to a least squares estimator1110 to estimate angular velocities cox and ω_(y) and translationalvelocity v_(z) ^(trans) over each of a series of time intervals in amanner similar to that described above during the first phase. In thisphase, calculated values of v_(x) and v_(y) for each point based onT_(i,i+1) ⁽¹⁾ as determined during the prior phase are input into leastsquares estimator 1120 as opposed to the feature measurements usedabove.

The primary difference between the third phase and the second phase isthat least squares estimator 1120 uses T_(i,i+1) ⁽¹⁾ to describe motionbetween the relevant frames 155. Least squares estimators 1110, 1120estimate new angular velocities ω_(x), ω_(y), and wand new translationalvelocities v_(x) ^(trans), v_(y) ^(trans), and v_(z) ^(trans) which maybe expressed as a transformation matrix T_(i,i+1) ⁽²⁾ for each of therelevant time intervals. Again, in some implementations of theinvention, a cumulative transformation matrix corresponding to the frameto frame time interval may be determined.

In various implementations of the invention, any of the phases of thefirst processing stage may be iterated any number of times as additionalinformation is gained regarding motion of target 190. For example, asthe transformation matrices are improved, each point 810 may be betterexpressed at a given reference time in relation to its measurement time.

During the first processing stage, the translational velocities of eachpoint (not otherwise available from the lidar measurements) may beestimated using features from the frames 155. Once all velocitycomponents are known or estimated for each point, transformationmatrices may be determined without using the feature measurements asillustrated in FIG. 13 .

FIG. 14 illustrates a block diagram of a configuration of processingsystem 160 that may be used during a first phase of the secondprocessing stage to refine a trajectory of target 190 according tovarious implementations of the invention. The first processing stageprovides transformation matrices sufficient to enable images 155 to bemapped onto images at any of the frame times. Once so mapped,differences in pixels themselves (as opposed to features in images 155)from different images transformed to the same frame time may be used tofurther refine the trajectory of target 190. In some implementations,various multi-frame corrections may be determined, which in turn throughthe least square estimator can be used for obtaining offsets between theconsecutive images Δx_(i,i+1), Δy_(i,i+1), Δθz_(i,i+1). Thesecorrections may be used to refine the transformation matrices (forexample, matrices T_(i,i+1) ⁽²⁾ to T_(i,i+1) ⁽³⁾). In someimplementations of the invention, during the first phase of the secondprocessing stage, a series of new transformation matrices (referred toherein as T_(i,i+1) ⁽³⁾) are refinements of T_(i,i+1) ⁽²⁾ on the basisof the offsets between image I_(i) and an image I_(j), namely, Δx_(i,j),Δy_(i,j), Δθz_(i,j). As illustrated, an estimator 1410 determines adifference between an image I_(i) and an image I_(j) using theappropriate transformation matrix T_(i,j) ⁽²⁾ to express the images atthe same frame time.

FIG. 15 illustrates a block diagram of a configuration of processingsystem 160 that may be used during a second phase of the secondprocessing stage to further refine a trajectory of target 190 accordingto various implementations of the invention. To the extent thatadditional accuracy is necessary, the transformation matrices from thefirst phase of the second processing stage (e.g., T_(i,i+1) ⁽³⁾) may beused in connection with the measurements from lidar subsystem 130 tofurther refine the transformation matrices (referred to herein asT_(i,i+1) ⁽⁴⁾). In some implementations of the invention, during thisphase, measurements from lidar subsystem 130 that occur within anyoverlap regions 360, 450 are used. These measurements correspond tomultiple measurements taken for the same point (or substantially thesame point) at different times. This phase is based on the premise thatcoordinate measurements of the same point on target 190 taken differenttimes should transform precisely to one another with a perfecttransformation matrix, i. e. points measured at the different times thathave the same x and y coordinates should have the same z coordinate. Inother words, all coordinates of the underlying points should directlymap to one another at the different points in time. The differences in zcoordinates (i.e., corrections) between the measurements at differenttimes can be extracted and input to the least square estimator. Throughthe least square estimator, the corrections to the transformationparameters between the consecutive frames may obtained and may beexpressed as Δz_(i,i+1), Δθx_(i,i+1), and Δθy_(i,i+1). These correctionsmay be used to refine the the transformation matrices T_(i,i+1) ⁽³⁾ toT_(i,i+1) ⁽⁴⁾. In some implementations of the invention, multiplemeasurements corresponding to use of an overscan beam may be used in asimilar manner.

In some implementations of the invention, factory calibration may beperformed to calibrate (i.e., register) measurements obtained from videosubsystem 150 with measurements obtained from lidar subsystem 130 asdescribed in, for example, U.S. patent application Ser. No. 13/839,216,entitled, “System and Method for Calibrating Video and LidarSubsystems,” filed on Mar. 15, 2013, and incorporated herein byreference as if reproduced below in its entirety. In someimplementations of the invention, field calibration, based on a target'sfacial features, may be performed to calibrate measurements obtainedfrom video subsystem 150 with measurements obtained from lidar subsystem130 as described in, for example, U.S. patent application Ser. No.13/840,833, entitled, “System and Method for Field Calibrating Video andLidar Subsystems using Facial Features,” filed on Mar. 15, 2013, andincorporated herein by reference as if reproduced below in its entirety.In some implementations, the video subsystem 150 and the lidar subsystem130 may be aligned with one another by mapping the measurements ofvarious facial features obtained by each of the subsystems to oneanother. In some implementations, various components of the lidarsubsystem 130 (e.g., scanning mirrors, etc.) are aligned with oneanother by mapping the measurements of various facial features obtainedby each of the components to one another.

In some implementations, beam steering mechanism 140 of the lidarsubsystem 130 may include a first scanning mirror and a second scanningmirror, each of which may move/scan independently from one another aswell as from video subsystem 150. In some implementations,three-dimensional measurements (also referred to as 3D point clouds, forexample, a point cloud 930) may be obtained using the first scanningmirror and the second scanning mirror. In some implementations,processing system 160 may receive the three-dimensional measurementsfrom lidar subsystem 130.

In some implementations, processing system 160 may compare a pluralityof first 3D point clouds (obtained using the first scanning mirror) witha plurality of second 3D point clouds (obtained using the secondscanning mirror. In some implementations, processing system 160 maydetermine differences in the locations (i.e., offsets) of a plurality offacial features between the first 3D point clouds and the second 3Dpoint clouds based on the comparison. In some implementations,processing system 160 may determine calibration adjustments to be madebetween the first and second scanning mirrors based on the determineddifferences. In other words, by comparing the 3D point clouds obtainedusing the first scanning mirror with the 3D point clouds obtained usingthe second scanning mirror, differences in the locations (e.g., offsets)of various facial features between the respective point clouds may beused to determine calibration adjustments between the first and secondscanning mirrors as would be appreciated.

In some implementations, the offsets (also referred to as ΔXYZ) betweenthe respective 3D point clouds may be determined at multiple ranges(i.e., distances) and/or locations. In some implementations, based onthese offsets at different scanning positions, the calibrationadjustments may be determined. In some implementations, the calibrationadjustments may be used to update/correct/adjust transformationparameters that would otherwise transform a coordinate system of thesecond 3D point clouds (obtained using the second scanning mirror) to acoordinate system of the first 3D point clouds (obtained using the firstscanning mirror) and vice versa. In effect, this calibrates the second3D point clouds obtained from the second scanning mirror to the first 3Dpoint clouds obtained from the first scanning mirror.

In some implementations, by determining offsets at different locationsor positions (providing sufficient variability in the measurements), afull set of transformation parameters may be updated, effectivelyaccounting for the translation and rotation of the two measurementsubsystems.

In some implementations, processing system 160 may align the lidarsubsystem 130 with the video subsystem (i.e., align the two subsystems)by subdividing the face and aligning the subsystems horizontally andvertically. In some implementations, for horizontal alignment betweenthe video system 150 and the lidar subsystem 130, processing system 160may determine a 3D symmetry plane dividing the face into left and righthalves based on the three-dimensional measurements (i.e., 3D pointclouds) obtained from the lidar subsystem 130. In some implementations,the 3D symmetry plane “cuts” the face through the tip of the nose,nose-bridge, lips, and forehead. In some implementations, processingsystem 160 may determine locations of symmetrical features of the face(namely, eye locations and nostril locations) based on images from thevideo subsystem 150. In some implementations, once the locations of thesymmetrical facial features are determined from the video system 150,processing system 160 may determine offsets in the locations of thesymmetrical facial features relative to the 3D symmetry plane (aspresumably, symmetrical features should be equidistant from the symmetryplane on the left and right sides thereof). In some implementations, thelocations of the symmetrical facial features may be determined using anynumber of 2D facial feature algorithms as would be appreciated. In someimplementations of the invention, a Fraunhofer face detection algorithmmay be used to detect eye positions, nostril positions, etc. In someimplementations, an eye detection algorithm from OpenCV may be used. Insome implementations of the invention, the averages of the locations ofthe facial features from different algorithms may be used as would beappreciated. For example, the locations of the eyes determined from theFraunhofer face detection algorithm may be averaged with the locationsof the eyes determined from the OpenCV eye detection algorithm. In someimplementations, these averages may be weighted depending on, forexample, a confidence associated with each of the algorithms.

In some implementation, the face symmetry plane may be inaccurate, sucha plane may be used to show the orientation of the face, but many facesare assymetrical by its nature. In some implementations, processingsystem 160 may locate/determine a corresponding noseridge in the 3Dpoint clouds from the lidar subsystem 130 as the relative maximum“ridge” between the two face halves, which would be true symmetry lineof the face. In some implementations, once the symmetrical face featureslike eyes and/or nostrils are determined from the video subsystem 150,the centers of these features are presumed to be on the noseridge. Insome implementations, the horizontal offset may be determined as theoffset between the center of the symmetrical features from the symmetryplane, corrected by the offset of the noseridge from the symmetry plane.

In some implementations, for vertical alignment between the video system150 and the lidar subsystem 130, processing system 160 may match alocation of the lip line between the video image of the video subsystem150 and the 3D point clouds of the lidar subsystem 130. On the videoimage, the lip line is typically the darkest point in the lip area,while in the 3D point cloud the lip line may be localized as the dipbetween the upper and lower lips. This “lip valley” determined from eachof the two subsystems may be used to accomplish vertical alignment(i.e., adjustments up and down to align the lip valley) and verticaloffsets and/or calibration adjustments between the two subsystems aswould be appreciated. In some implementations, processing system 160 maycompare the lip line estimate from the video subsystem with the lip lineestimate from the lidar subsystem. In some implementations, processingsystem 160 may determine one or more vertical offsets based on thecomparison. In some implementations, processing system 160 may determineone or more vertical calibration adjustments to be made between thelidar subsystem and the video subsystem based on the vertical offsets.

In some implementations, the calibration offset between the 3D pointclouds and the video image can be found from the irises misalignments.Particularly, the laser is mostly absorbed by irises, producing thesignal with low SNR, which allows their precise 3D (lidar)identifications. In most cases, the iris location can also be found fromthe video image by finding the maxima of the gradient semi-circleintensity. In some implementations, lidar video misalignments betweenthe irises may provide the horizontal and vertical offsets between thelidar and video subsystems, which can be used solely or in addition tothe offsets from the symmetry plane and the lip lines. This is discussedin further detail below.

Other facial features may be used in addition to those discussed aboveas would be appreciated. For example, the nosetip as determined fromboth the video image and the 3D point cloud may also be used to provideaspects of both vertical and horizontal alignment as would beappreciated.

The vertical offsets and the horizontal offsets correspond to thedifferences between the video image and the 3D point cloud. Bycalculating such offsets for multiple ranges and positions, thetransformation parameters (for example projective transformations)between the coordinate cells of the video subsystem and the lidarsubsystem may be adjusted/updated, thereby aligning/calibrating thesetwo subsystems with one another as would be appreciated.

The calibration procedures discussed above may be performed continuouslyin the background during operation of the combined video/lidar system100, at periodic intervals during its operation or at any time/eventspecified by an operator as would be appreciated.

FIG. 16 illustrates facial features monitored by the lidar subsystem andthe video subsystem according to various implementations of theinvention. In some implementations, the facial features depicted in FIG.16 may be used for calibrating the video subsystem 150 and the lidarsubsystem 130. In some implementations, FIG. 16 depicts the followingexemplary facial features determined based on images from the videosubsystem 150: eyes (designated as Eyes F in FIG. 16 ), an estimate ofthe nosebridge based on the eyes F (designated as NB F in FIG. 16 ),estimate of the nosebridge based on eye detection algorithm from OpenCV(designated as NB Video in FIG. 16 ), an OpenCV estimate of the nose tip(designated as OCV Nose Tip in FIG. 16 ), an OpenCV estimate of themouth center (designated as OCV Mouth Center in FIG. 16 ), an estimateof the individual nostrils (designated as Nostrils in FIG. 16 ), acenter of the nostrils (designated as Nostril Center in FIG. 16 ), and avideo estimate of vertical lip line position (for multipleranges)-looking for changes in video luminance (designated as Video LipLine in FIG. 16 ).

In some implementations, FIG. 16 further depicts the following exemplaryfacial features determined based on 3D measurements/3D lidar data/3Dpoint clouds from the lidar subsystem 130: an estimate of the nosebridge(designated as NB Lidar in FIG. 16 ), an estimate of the nose ridge(designated as NR Lidar in FIG. 16 ), nose ridge extracted from the facesymmetry line (designated as NR Sym Line in FIG. 16 ), and a lidarestimate of vertical lip line position (for multipleranges)-“valley-processing” (designated as Lidar Lip Line in FIG. 16 ).

FIG. 17 illustrates an exemplary lidar-based face profile (i.e., faceprofile based on the 3D point clouds from the lidar subsystem 130)according to various implementations of the invention. FIG. 17 depictsthe face profile at the symmetry line (designated as Symmetry Line), thesmoothed symmetry line at the lip area (designated as Lip Area), theupper and lower lips (designated as Lips), and the lip line (designatedas Lip Line).

FIG. 18 illustrates a flowchart depicting example operations performedby a system for calibrating a lidar subsystem, according to variousaspects of the invention. In some implementations, the describedoperations may be accomplished using one or more of the componentsdescribed herein. In some implementations, various operations may beperformed in different sequences. In other implementations, additionaloperations may be performed along with some or all of the operationsshown in FIG. 18 . In yet other implementations, one or more operationsmay be performed simultaneously. In yet other implementations, one ormore operations may not be performed. Accordingly, the operationsdescribed in FIG. 18 and other drawing figures are exemplary in natureand, as such, should not be viewed as limiting.

In some implementations, in an operation 1802, process 1800 may comparea plurality of first 3D point clouds (obtained using the first scanningmirror) with a plurality of second 3D point clouds (obtained using thesecond scanning mirror). In some implementations, in an operation 1804,process 1800 may determine differences in the locations (i.e., offsets)of a plurality of facial features between the first 3D point clouds andthe second 3D point clouds based on the comparison. In someimplementations, in an operation 1806, process 1800 may determinecalibration adjustments to be made between the first and second scanningmirrors based on the determined differences. In other words, bycomparing the 3D point clouds obtained using the first scanning mirrorwith the 3D point clouds obtained using the second scanning mirror,differences in the locations (e.g., offsets) of various facial featuresbetween the respective point clouds may be used to determine calibrationadjustments to the transformation parameters based on the determineddifferences as would be appreciated.

FIG. 19 illustrates a flowchart depicting example operations performedby a system for calibrating a lidar subsystem and a video subsystem,according to various aspects of the invention. In some implementations,in an operation 1902, operation 1900 may determine locations ofsymmetrical features of the face (namely, eye locations and nostrillocations) based on images from the video subsystem 150 as well as the3D face symmetry plane from the point cloud. In some implementations,the 3D symmetry plane may be determined by dividing the face into leftand right halves based on the three-dimensional measurements (i.e., 3Dpoint clouds) obtained from the lidar subsystem 130. In someimplementations, in an operation 1904, operation 1900 may determine thenoseridge and the noseridge offset relative to the 3D symmetry plane.

In some implementations, in an operation 1906, process 1900 maydetermine a horizontal offset between the video symmetrical points (i.e.locations of the plurality of symmetrical features) and the 3D symmetryplane. In some implementations, in an operation 1908, process 1900 maycorrect the horizontal offset by the shift between the 3D symmetry planeand the noseridge.

In some implementations, in operation 1910, process 1900 may determinethe vertical offset between the lidar and video lip line alignments. Insome implementations, process 1900 may determine a first location of lipline based on the images and a second location of the lip line based onthe point cloud. In some implementations, process 1900 may determine thevertical offset based on the first and second locations.

In some implementations, in an operation 1912, process 1900 maydetermine calibration adjustments to the projective transformations(transformation parameters) based on determined video and lidar offsets(i.e., horizontal and vertical offsets).

As mentioned above, in some implementations, a calibration offsetbetween the 3D point clouds and the video image (i.e., calibrationoffsets between lidar subsystem 130 and video subsystem 150) may bedetermined from misalignments in location of the irises. Laser light isabsorbed by irises to a greater degree than the other tissues of the eyeor surrounding face. Thus, return signals from the iris(es) areattenuated in relation to return signals from other surrounding tissuesand detected return signals from the irises have a lower signal-to-noiseratio, amplitute, etc. By locating and identifying those 3D points inthe point cloud in the region of the eye that have lower signal-to-noiseratios than their surrounding 3D points, the iris may be located bylidar susbsystem.

In some implementations of the invention, because of scanning, lidarsubsystem 130 “samples” a region of the eye, and hence, does not providea sufficient number of points to clearly map out the iris. In theseimplementations, the locations of the points that have an attenuatedreflected signal in the region of the eye may be averaged to provide anaverage location, or centroid location, of such points which may be usedto approximate a centroid of the iris. In some implementations of theinvention, lidar subsystem 130 may focus a scan in the region of the eyeto gain additional samples of the eye and fully sample the iris as wouldbe appreciated.

FIG. 20 illustrates an image 2010 from video subsystem 150 on which isoverlayed a number of points 2020 (illustrated as dark dots in FIG. 21 )including a subset 2030 of points 2020 (illustrated as light dots inFIG. 21 and referred to herein as “iris points 2030) measured by lidarsubsystem 130 in accordance with various implementations of theinvention. Iris points 2030 correspond to those points 2020 that have anattenuated reflected signal relative to other points 2020. Thisattenuated reflected signal of iris points 2030 may be discriminatedfrom such other points 2020 in the region of the eye as being pointscorresponding to the iris. For example, points 2020 having an attenuatedreflected signal of a threshold 10 dB or more in relation to otherpoints 2020 in the region of the eye may be identified as iris points2030. Other threshold values may be used depending on factors associatedwith lidar subsystem 130, etc., and may be set based on experimentationas would be appreciated.

FIG. 21 illustrates a relevant portion 2110 of the image 2010illustrated in FIG. 20 in further detail. In particular, FIG. 21illustrates a centroid 2120 of iris points 2030, which as discussedabove, may be determined as an average location of the iris points 2030.FIG. 21 also illustrates an image iris 2130 as a dashed line around anapproximate location of the iris in image 2010 (solely for purposes ofillustration). As illustrated in FIG. 21 , iris points 2030 are offsetfrom image iris 2130 as a result of the misalignment between lidarsubsystem 130 and video subsystem 150.

FIG. 22 illustrates a relevant portion 2210 of the image 2010illustrated in FIG. 20 in further detail. In particular, FIG. 22illustrates an iris contour 2230 and a centroid 2220 and a centroid 2240as determined from image 2010 via various facial recognition techniquesas would be appreciated. In some implementations of the invention,centroid 2220 may be determined as an average location of points in iriscontour 2230 as would be appreciated. In some implementations of theinvention, centroid 2240 may be determined using variouswell-established Fraunhofer facial recognition processing techniques aswould be appreciated. In various implementations of the invention,either centroid 2220 or centroid 2240 may be used as a centroid of theiris. In various implementations of the invention, an average (weightedor otherwise) of centroids 2220, 2240 and/or other centroids determinedfrom image 2010 may be used as the centroid of the iris.

As illustrated in FIG. 22 , a misalignment (i.e., calibration offset)exists in between centroid 2120 determined from lidar subsystem 130 andcentroids 2220, 2240 determined from video subsystem 150. Thiscalibration offset is apparent in FIG. 21 as the iris points 2030 fromlidar subsystem 130 are not mapped onto a face 2040 of image 2010 whereone would expect them. For example, one would expect iris points 2030 tobe mapped further to the right and slightly up from their position inFIG. 21 . This “mismapping” is a result of the misalignment betweenlidar subsystem 130 and video subsystem 150.

FIG. 22 also illustrates an offset between a location of an iris basedon the image from the video subsystem and a location of the iris basedon the measurements from the lidar subsystem. In other words, FIG. 22illustrates the offset between centroid 2120 determined frommeasurements from lidar subsystem 130 and centroid 2220, 2240 determinedfrom images from video subsystem 150. As discussed above, this offsetmay be used to correct the alignment between lidar subsystem 130 andvideo subsystem 150.

FIG. 23 illustrates an operation of processing system 160 to determinecalibration offsets between lidar subsystem 130 and video subsystem 150in accordance with various implementations of the invention. In anoperation 2310, processing system 160 receives a plurality ofmeasurements of a target generated by lidar subsystem 130. In anoperation 2320, processing system 160 receives an image of a target fromvideo subsystem 150. In some implementations of the invention, a time ofthe image of the target matches closely to a time(s) when lidarsubsystem 130 was scanning an eye or iris region of the target; hencethe plurality of measurements generated by lidar system coincide closelyin time with the image of the target. In an operation 2330, processingsystem 160 determines a centroid (or other suitable location) of an irisof the target based on the plurality of measurements from lidarsubsystem 130. As discussed above, in some implementations of theinvention, this may be accomplished by discriminating the measurementsbased on an amplitude of a signal reflected back from the target. In anoperation 2340, processing system 160 determines a centroid (or othersuitable location) of an iris of the target based on the image fromvideo subsystem 150. In an operation 2350, processing system 160determines an offset between the centroid of the iris determined fromthe measurements from lidar subsystem 130 and the centroid of the irisdetermined from the image from video subsystem 150. In an operation2360, processing system 160 calibrates lidar subsystem 130 with videosubsystem 150 based on the offset between the respective centroids. Insome implementations of the invention, calibration between lidarsubsystem 130 and video subsystem may be performed based scans andimage(s) at a specific period of time or over multiple periods of time(i.e., as a result of revisiting or rescanning the eye/iris region ofthe target) as would be appreciated.

While the invention has been described herein in terms of variousimplementations, it is not so limited and is limited only by the scopeof the following claims, as would be apparent to one skilled in the art.These and other implementations of the invention will become apparentupon consideration of the disclosure provided above and the accompanyingfigures. In addition, various components and features described withrespect to one implementation of the invention may be used in otherimplementations as well.

What is claimed is:
 1. A method for calibrating a lidar subsystem with avideo subsystem, the method comprising: receiving a plurality ofmeasurements generated by the lidar subsystem, each of the plurality ofthree-dimensional measurements corresponding to a point on a target,each of the plurality of measurements corresponding to a location of thepoint on the target and an amplitude of a return signal from the pointon the target; receiving at least one image of the target generated bythe video subsystem; determining a first location of an iris of thetarget based on the amplitude of a return signal of at least some of theplurality of three-dimensional measurements; determining a secondlocation of the iris of the target based on the image of the target;determining an offset between the first location of the iris and thesecond location of the iris; and calibrating the lidar subsystem withthe video subsystem based on the offset.
 2. The method of claim 1,wherein a location of the point on the target comprises an azimuth, anelevation, and a range.
 3. The method of claim 2, wherein the locationof the point on the target further comprises a Doppler velocity.
 4. Themethod of claim 1, wherein determining a first location of an iris ofthe target based on the amplitude of the return signal of at least someof the plurality of measurements comprises: determining which of theplurality of three-dimensional measurements have a low amplitude of thereturn signal relative to other of the plurality of measurements.
 5. Themethod of claim 4, wherein determining a first location of an iris ofthe target based on the amplitude of the return signal of at least someof the plurality of three-dimensional measurements further comprises:for those of the plurality of measurements that have the low amplitudeof the return signal, determining a centroid location from therespective locations of those of the plurality of measurements; andwherein the first location is the centroid location.
 6. The method ofclaim 1, wherein determining a second location of the iris of the targetbased on the image of the target comprises: determining an iris contourof the iris of the target based on the image of the target.
 7. Themethod of claim 6, wherein determining the second location of the irisof the target based on the image of the target further comprises:determining a centroid location from the iris contour; and wherein thesecond location is the centroid location.
 8. The method of claim 1,wherein determining a second location of the iris of the target based onthe image of the target comprises: performing Fraunhofer processing onthe image of the target to identify a centroid location of the iris ofthe target; and wherein the second location is the centroid location. 9.A system for calibrating a lidar subsystem with a video subsystem, thesystem comprising: a lidar subsystem configured to generate a pluralityof measurements, each of the plurality of measurements corresponding toa point on a target, each of the plurality of measurements comprising alocation of the point on the target and an amplitude of a return signalfrom the point on the target; a video subsystem configured to provide animage of the target; and a processor configured to: receive theplurality of measurements from the lidar subsystem, receive the at leastone image of the target from the video subsystem, determine a firstlocation of an iris of the target based on the amplitude of the returnsignal of at least some of the plurality of measurements, determine asecond location of the iris of the target based on the image of thetarget, determine an offset between the first location of the iris andthe second location of the iris, calibrate the lidar subsystem with thevideo subsystem based on the offset.
 10. The system of claim 9, whereina location of the point on the target comprises an azimuth, anelevation, and a range.
 11. The system of claim 10, wherein the locationof the point on the target further comprises a Doppler velocity.
 12. Thesystem of claim 9, wherein the determine a first location of an iris ofthe target based on the amplitude of the return signal of at least someof the plurality of three-dimensional measurements comprises: determinewhich of the plurality of measurements have a low amplitude relative toother of the plurality of measurements.
 13. The system of claim 12,wherein the determine a first location of an iris of the target based onthe amplitude of the return signal of at least some of the plurality ofmeasurements further comprises: for those of the plurality ofmeasurements that have the low amplitude, determine a centroid locationfrom their respective locations, and wherein the first location is thecentroid location.
 14. The system of claim 9, wherein determine a secondlocation of the iris of the target based on the image of the targetcomprises: determine an iris contour of the iris of the target based onthe image of the target.
 15. The system of claim 14, wherein determinethe second location of the iris of the target based on the image of thetarget further comprises: determine a centroid location from the iriscontour, and wherein the second location is the centroid location. 16.The system of claim 1, wherein determine a second location of the irisof the target based on the image of the target comprises: performFraunhofer processing on the image of the target to identify a centroidlocation of the iris of the target; and wherein the second location isthe centroid location.